On Characterization of Entropy Function via Information Inequalities

Given n discrete random variables = fX1; ; Xng, associated with any subset of f1; 2; ; ng, there is a joint entropy H(X ) where X = fXi: i 2 g. This can be viewed as a function defined on 2 2; ; ng taking values in [0; +1). We call this function the entropy function of . The nonnegativity of the joint entropies implies that this function is nonnegative; the nonnegativity of the conditional joint entropies implies that this function is nondecreasing; and the nonnegativity of the conditional mutual informations implies that this function has the following property: for any two subsets and of f1; 2; ; ng H ( ) +H ( ) H ( [ ) +H ( \ ): These properties are the so-called basic information inequalities of Shannon’s information measures. Do these properties fully characterize the entropy function? To make this question more precise, we view an entropy function as a 2 1-dimensional vector where the coordinates are indexed by the nonempty subsets of the ground set f1; 2; ; ng. Let n be the cone in R 1 consisting of all vectors which have these three properties when they are viewed as functions defined on 2 2; ; . Let n be the set of all 2 1-dimensional vectors which correspond to the entropy functions of some sets of n discrete random variables. The question can be restated as: is it true that for any n, n = n? Here n stands for the closure of the set n. The answer is “yes” when n = 2 and 3 as proved in our previous work. Based on intuition, one may tend to believe that the answer should be “yes” for any n. The main discovery of this paper is a new information-theoretic inequality involving four discrete random variables which gives a negative answer to this fundamental problem in information theory: n is strictly smaller than n whenever n > 3. While this new inequality gives a nontrivial outer bound to the cone 4, an inner bound for 4 is also given. The inequality is also extended to any number of random variables.